Augmented reality systems project virtual characters and objects into physical locations, allowing for immersive experiences and novel interaction models. In particular, in some augmented reality systems, virtual characters or objects may be inserted into real-world images, for example by overlaying a captured image or video stream with a two- or three-dimensional animated rendering of the virtual character. In some systems, a physical object recognized in the captured image may be replaced by a virtual object associated with that physical object. For example, recognized vehicles in the captured image may be recognized and replaced with animated vehicles (e.g., military vehicles, cartoon vehicles, etc.).